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Objective: The aim of the study reported here was to validate the risk-scoring algorithm for 
prognostication of scrub typhus severity. 

Methods: The risk-scoring algorithm for prognostication of scrub typhus severity developed 
earlier from two general hospitals in Thailand was validated using an independent dataset of 
scrub typhus patients in one of the hospitals from a few years later. The predictive performances 
of the two datasets were compared by analysis of the area under the receiver-operating charac- 
teristic curve (AuROC). Classification of patients into non-severe, severe, and fatal cases was 
also compared. 

Results: The proportions of non-severe, severe, and fatal patients by operational definition 
were similar between the development and validation datasets. Patient, clinical, and laboratory 
profiles were also similar. Scores were similar in both datasets, both in terms of discriminating 
non-severe from severe and fatal patients (AuROC =88.74% versus 91.48%, P=0.324), and in 
discriminating fatal from severe and non-severe patients (AuROC =88.66% versus 91.22%, 
P=0.407). Over- and under-estimations were similar and were clinically acceptable. 
Conclusion: The previously developed risk-scoring algorithm for prognostication of scrub 
typhus severity performed similarly with the validation data and the first dataset. The scoring 
algorithm may help in the prognostication of patients according to their severity in routine 
clinical practice. Clinicians may use this scoring system to help make decisions about more 
intensive investigations and appropriate treatments. 
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Introduction 

"Scrub typhus," one of several potentially fatal tropical rickettsial infections, is caused 
by Orientia tsutsugamushi and presents as a systemic, vasculitis-like infection. Clinical 
manifestations are typically fever, eschar, generalized or regional lymphadenopathy, 
maculopapular rash, severe headache, and myalgia. Nausea and vomiting, diarrhea, 
and conjunctival injection can also occur. 1 - 2 Without appropriate treatment, infected 
patients usually experience complications in the second week. Those especially at risk 
of infection are the elderly, those without previous immunity, 2 " 4 and those with risk 
factors such as hypoalbuminemia, hyperbilirubinemia, high levels of aspartate amino- 
transferase (AST) and/or creatinine, and those with lung crepitations. 3,5-10 Patients with 
complications who are admitted to intensive care units with a high Acute Physiology 
and Chronic Health Evaluation II (APACHE II) score are at high risk of mortality. 5 
In rural areas where effective treatment is unavailable or delayed, mortality rates have 



been reported as high as 35° 
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The cost of treatment for scrub typhus patients with 
complications is much higher due to hospital admissions 
and more intensive treatments. 11 Prognostication of disease 
severity may help with the identification of disease severity 
and prompt initiation of treatments. 

Prognostication of disease severity has been considered 
by researchers of various conditions. Examples include a 
clinical prediction rule for severe community-acquired pneu- 
monia, 12 Mortality in Emergency Department Sepsis score, 13 
a prediction rule for risk of mortality in Clostridium difficile 
infection, 14 and decision-tree algorithms for dengue fever 15 and 
dengue hemorrhagic fever. 16 Some of these tools were found 
to have both advantages and disadvantages when validated. 

Previously we developed a clinical risk-scoring algorithm 
for scrub typhus severity comprising six predictors from 
routine practice. 17 The objective of the study reported here 
was to validate the risk-scoring algorithm developed earlier 
with an independent dataset obtained from similar patients 
from a few years later. 

Materials and methods 

Patients 

Data were retrieved retrospectively from the medical files 
of scrub typhus patients. The criteria for inclusion were 
similar to those used in the previous study 17 and were based 
on history of disease exposure and presentation with acute 
fever and at least one of the following signs and symptoms: 
myalgia, headache, conjunctival injection, cough, profuse 
sweating, maculopapular rash, and/or lymphadenopathy 
accompanied by the presence of eschar, and/or a positive 
immuno-chromatographic test for scrub typhus. 18 
Patients were classified into three groups: 

1 . non-severe - patients without any complications 

2. severe - patients with complications involving at least 
one of the following organ systems: 

• cardiovascular system - presence of any of any of the 
following: systolic blood pressure <90 mmHg, myo- 
carditis (defined as elevated creatine kinase-myocardial 
band isoenzyme above baseline), or abnormal cardiac 
arrhythmia with no previous history of atrial fibril- 
lation, supra ventricular tachycardia, or frequent 
premature ventricular tachycardia 

• respiratory system - presence of acute respiratory 
distress syndrome, defined as the ratio of partial pres- 
sure arterial oxygen and fraction of inspired oxygen 
(Pa0 2 /Fi0 2 ) <200 mmHg, with bilateral interstitial 
infiltration on chest film with normal cardio/thoracic 
ratio, or no volume overload of central venous pres- 
sure from central venous catheter 



• central nervous system - presence of any of the 
following: Glasgow Coma Scale ^12 without 
other causes, seizure without other causes, or 
meningoencephalitis 

• hematological system - platelet count <20,000/mm 3 

• urinary system - presence of acute renal failure, 
defined as creatinine s2mg/dL or creatinine change 
of >0.5 mg/dL/day 

• gastrointestinal and hepatobiliary system - presence 
of hepatitis, defined as elevated AST or alanine amino- 
transferase (ALT) more than fivefold normal levels. 

3. fatal - patients who died in hospital from scrub typhus. 

Patients in any intervention trials during the same period 
were excluded from the data analysis. 

Development data 

The development data were for patients admitted to two 
university-affiliated referral hospitals - Nakornping Hospital, 
Chiang Mai and Chiangrai Prachanukroh Hospital - in the 
north of Thailand from 2004 to 2010 (n=526). 

Validation data 

The validation data were for patients admitted to Chiangrai 
Prachanukroh Hospital from 201 1 to 2012 (n=257). 

Data analysis 

The characteristics of the development data and the valida- 
tion data were compared using an exact probability test, 
Student's Mest, or Wilcoxon rank-sum test as appropriate. 
The patient risk scores were assigned following the scoring 
scheme previously developed (Table l). 17 Patients were 
categorized into three severity groups: 1) non-severe, 2) 
severe, and 3) fatal, as described. The predictive perfor- 
mance of the scores was compared by analysis of the area 
under the receiver-operating characteristic curve (AuROC). 



Table 1 Significant predictors 


and assigned item 


scores 


Predictor 


Category 


Assigned score 


Age (years) 


<I5 


0 




>IS 


3 


Pulse rate (/minute) 


=£100 


0 




> 100 


2 


Crepitation 


No 


0 




Yes 


2 


AST (IU/L) 


£160 


0 




> 160 


2 


Serum albumin (g/dL) 


<3.0 


3 




>3.0 


0 


Serum creatinine (mg/dL) 


<l.4 


0 




>l.4 


4 



Abbreviation: AST, aspartate aminotransferase. 
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Over- and underestimated proportions were calculated and 
compared. 

Ethical approval 

The study was approved by the Ethics Committee for 
Research in Patients, Faculty of Medicine, Chiang Mai 
University, and the research ethical committees of the two 
hospitals. 

Table 2 Demographic and clinical characteristics of scrub typhus 
patients in the development and validation datasets 



Table 3 Laboratory test findings 



Characteristics 


Development, 


Validation, 


P-value 




n=526 (n [%]) 


n=zb / (n \_/o}) 




Demographic and history 








Male 


291 (55.3) 


165 (64.2) 


0.021 


Age (years) 


29.3±22.0 


3 1 .0+2 1 .4 


0.1 15 


(mean ± SD) 








Underlying diseases 


52 (9.9) 


23 (8.9) 


0.796 


Fever duration (days) 


6.5±3.2 


6.6±3.8 


0.151 


(mean ± SD) 








Length of hospital 


4.1 ±2.6 


4.8±3.9 


0.246 


stay (days) (mean ± SD) 








Definition-classified severity 






Non-severe 


357 (67.9) 


1 60 (62.3) 


0.219 


Severe 


100 (19.0) 


62 (24. 1 ) 




Fatal 


69 (13.1) 


35 (13.6) 




Clinical characteristic 








Headache 


160 (30.4) 


96 (37.5) 


0.062 


Myalgia 


105 (20.0) 


61 (23.7) 


0.228 


Conjunctival injection 


484 (92.0) 


185 (72.0) 


<0.00l 


Maculopapular rash 


485 (92.2) 


187 (72.8) 


<0.00l 


Lymphadenopathy 


373 (70.9) 


154 (59.9) 


0.003 


Eschar 


290 (55.1) 


41 (15.6) 


<0.00l 


Abdominal pain 


178 (33.8) 


65 (25.3) 


0.017 


Nausea, vomiting 


122 (23.2) 


75 (29.2) 


0.079 


Diarrhea 


102 (19.4) 


38 (14.8) 


0.136 


Hepatomegaly 


163 (31.0) 


67 (26.1) 


0.181 


Splenomegaly 


53 (10.1) 


20 (7.8) 


0.360 


Jaundice 


49 (9.3) 


20 (7.8) 


0.506 


Seizure 


26 (4.9) 


15(5.8) 


0.61 1 


Stiff neck 


14(2.7) 


17 (6.6) 


0.01 1 


Crepitation 


56 (10.7) 


44 (17.1) 


0.012 


Wheezing 


15 (2.9) 


5 (2.0) 


0.630 


Cough 


169 (32.1) 


88 (34.2) 


0.571 


Dyspnea 


39 (7.4) 


29 (1 1.3) 


0.079 


Respiration (/minute) 


25.4±8.8 


25.3±I0.5 


0.964 


(mean ± SD) 








Pulse (/minute) 


I04.0±22.l 


102.1 ±20.4 


0.237 


(mean ± SD) 








SBP (mmHg) 


I02.8±I7.6 


I06±2I.0 


0.012 


(mean ± SD) 








DBP (mmHg) 


63.4±I2.I 


66.5±I5.7 


0.005 


(mean ± SD) 








Body temperature (°C) 


38.2±l.2 


38.3±2.9 


0.496 


(mean ± SD) 









Characteristics 


Development, 


Validation, 


P-value 




n=526 


n=257 






(mean x auj 


(mean x jU ) 




Hematological 








WBC (xlOVmm 3 ) 


9.9±5.4 


I0.9±I0.6 


0.288 


Neutrophils (%) 


70.2±I7.I 


7I.3±I5.8 


0.387 


Lymphocytes (%) 


2I.I±I4.6 


20.1 + 1 3.6 


0.525 


Monocytes (%) 


6.1 ±7.7 


6.0±6.3 


0.464 


Platelets (x 1 0 3 /mm 3 ) 


I33.I±I07.8 


I67.2± 127.5 


<0.00l 


Hematocrit (%) 


35.2±6.2 


36.5±6.3 


0.498 


Hemoglobin (g/dL) 


1 1.8+2.1 


12.1 ±2.2 


0.156 


Biochemistry 








BUN (mg/dL) 


27.6±26.7 


26.7±26.4 


0.681 


Creatinine (mg/dL) 


1.7+1.9 


l.8±2.l 


0.431 


Total bilirubin (mg/dL) 


2.6±4.4 


l.9±2.5 


0.348 


Direct bilirubin (mg/dL) 


1 .3+2.3 


0.8±l.2 


0.175 


Albumin (g/dL) 


2.7±0.6 


3.1+0.7 


0.014 


Globulin (g/dL) 


3.3±0.7 


3.4±0.7 


0.063 


ALT (IU/L) 


I25.2±259.7 


143.1 ±505.3 


<0.00l 


AST (IU/L) 


2I5.0±459.5 


375.1 ±2,528.4 


<0.00l 


ALP (IU/L) 


25 1 .3+1 93.5 


20 1.5± 164.6 


<0.00l 


Na (mmol/L) 


I33.3±4.7 


I34.0±I0.8 


0.265 


K (mmol/L) 


3.6±0.6 


3.8±2.l 


0.028 


Cl 2 (mmol/L) 


I00.7±5.6 


I0I.4±6.0 


0.163 


C0 2 (mmol/L) 


2I.2±4.7 


2I.8±5.2 


0.121 


Urine albumin (n [%]) 








<l + 


242 (54.6) 


1 18 (51.5) 


0.463 


2:1 + 


201 (45.4) 


1 1 1 (48.5) 




Urine sugar (n [%]) 








<l + 


435 (98.2) 


223 (97.4) 


0.571 


2:1 + 


8(1.8) 


6 (2.6) 





Notes: The data were missing for some parameters: in the development set, urine 
albumin and urine sugar -83 cases; in the validation set, AST —42 cases, albumin —68 cases, 
creatinine -48 cases, and urine albumin and urine sugar —28 cases. 
Abbreviations: ALP, alkaline phosphatase; ALT, alanine aminotransferase; AST, 
aspartate aminotransferase; BUN, blood urea nitrogen; Cl 2 , chloride; C0 2 , carbon 
dioxide; K, potassium; Na, sodium; SD, standard deviation; WBC, white blood cell. 



Results 

Data for 257 patients comprised the validation dataset. Of 
those, 160 cases were defined as non-severe, 62 as severe, and 
35 as fatal. The distribution of patients into the three severity 
groups of the development and the validation datasets was 
similar. The clinical characteristics of the patients were also 

Table 4 Score-classified scrub typhus severity levels in the 
development and validation datasets 



Abbreviations: DBP, diastolic blood pressure; 
standard deviation. 



SBP, systolic blood pressure; SD, 



Score-classified 


Development 


Validation 


P-value 


severity levels 


(n=S26) 


(n=l67) 




Mean score ± SD 


6.0±3.9 


7.1 ±3.9 


0.002 


Severity levels (n, %) 








Non-severe 


278 (52.8) 


64 (38.3) 


0.004 


Severe 


143 (27.2) 


56 (35.5) 




Fatal 


105 (20.0) 


47 (28.1) 





Abbreviation: SD, standard deviation. 
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Table 5 Score-classified severity levels, criterion-c 


lassified severity levels, 


and risk estimation validity 






Score-classified 
severity levels 


Score 
range 


Criterion-classified severity levels 




Risk estimation validity* 




Non-severe, n=78 


Severe, n=55 


ratal, n=jl 


/— , ...... /o/\ 

Over (7o) 


Correct (%) 


Under (%) 


Mean ± SD 




4.4±2.4 


8.4±3.4 


1 1 .4±2.S 








IQR 




3-6 


7-12 


9-14 








Non-severe, n=64 


<5 


53 


1 1 


0 




31.7 


6.6 


Severe, n=S6 


6-9 


23 


24 


9 


13.8 


14.4 


5.4 


Fatal, n=47 


>I0 


2 


20 


25 


13.2 


14.9 




Total 










27.5 


61.0 


12.0 



Note: *Percentage of total patients. 

Abbreviations: IQR, interquartile range; SD, standard deviation. 



similar except in terms of 1) sex, 2) symptoms or signs of 
conjunctival injection, maculopapular rash, lymphadenopathy, 
eschar, abdominal pain, stiff neck, and crepitation, 3) vital 
signs of systolic blood pressure, and diastolic blood pressure 
(Table 2). The laboratory investigation data also showed simi- 
lar profiles for both datasets, except in terms of platelet count 
and AST, alanine aminotransferase, alkaline phosphatase, 
albumin, and potassium levels (Table 3). 

The risk score was assigned to patients with six complete 
clinical parameters (n=167, 65%). The mean risk scores of 
patients in the validation data were different to those of patients 
in the development dataset (mean + sd 7.1+3.9 and 6.0+3.9, 
respectively). The proportion of patients classified into the 
three severity levels was also different (P=0.004; Table 4). 

In the validation dataset, the score correctly predicted 
31.7% (53 out of 78) of patients into the non-severe group 
(score ^5) with one-level underestimation of 6.6% (eleven 
patients) and no two-level underestimation, while 14.4% 
(24 out of 55) of patients were correctly classified into the 
severe group (score 6-9), with an underestimation of 5.4% 
(nine patients) and an overestimation of 13.8% (23 patients). 
Finally, for the fatal severity group (score ^10), the score 
was predicted correctly for 14.9% (25 of 34) of patients, 
with one-level overestimation of 12.0% (20 patients) and 
two-level overestimation of 1.2% (2 patients), giving a total 
overestimation of 13.2% (Table 5). 

The overall predictive performance (as assessed by 
the AuROCs) of the validation dataset was insignificantly 
lower than that of the development dataset, both in the dis- 
crimination between non-severe versus (vs) severe and fatal 



(AuROC =88.74% vs 91.48%, P=0.324) and between fatal 
versus non-severe and severe (AuROC =88.66% vs 91.22%, 
P=0.407; Table 6 and Figure 1). 

Discussion 

Validation is needed for any clinical prediction rules before 
they can be adopted in clinical practice, either internally or 
externally. The Mortality in Emergency Department Sepsis 
score, for example, was developed to evaluate the risk of 
death in patients in emergency departments, in which valida- 
tion in one-third of the sample yielded similar discrimination 
(AuROC =0.82 vs 0.78). 13 Further, a clinical prediction rule 
for severe community-acquired pneumonia was devel- 
oped to screen patients for initial treatments in emergency 
departments. The rule performed similarly to the develop- 
ment data when prospectively validated in patients from the 
same hospitals (AuROC =0.92). However, the performance 
was poorer when externally validated in patients from other 
hospitals (AuROC =0.80). 12 External validation of a predic- 
tion rule for risk of mortality in Clostridium difficile infection 
also yielded poorer performance (AuROC =0.704 vs 0.653). 14 
It is very common for the performance of the validation data 
to be poorer than that of the development data. 

Our clinical risk-scoring algorithm for severe scrub typhus 
was validated against patients admitted in only one hospital 
some years after the patients used in the development dataset. 
When only patients with complete information on the six 
predictors (n=167) were selected from the validation dataset, 
the performance of the algorithm was somewhat poorer than 
with the development data, both in terms of differentiating 



Table 6 Discriminative performance of scrub typhus severity score in the development and validation datasets 

Prediction/discrimination Development (n=526) Validation (n= 1 67) P-value 

AuROC (%) 95% CI AuROC (%) 95% CI 

Non-severe versus severe + fatal 91.48 88.58-94.10 88.74 83.78-93.71 0.324 

Non-severe + severe versus fatal 91.22 88.23-94.20 88.66 83.52-93.79 0.407 

Abbreviations: AuROC, area under the receiver-operating characteristic curve; CI, confidence interval. 
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non-severe from severe and fatal (AuROC =88.74% vs 
91.48%), and fatal from non-severe and severe (AuROC 
=88.66% vs 91.22%). However, these differences were not 
statistically or clinically significant. The overall proportion of 
correct classifications was somewhat lower (6 1 .0% vs 68.3%), 
but overestimation was higher (27.0% vs 25.8%), which may 
be considered more beneficial to the patients. 17 

There were incomplete data on important predictors 
for 90 of 257 patients (35%). The missing data were AST 
(n=42, 24.1%), albumin (n=68, 26.5%), and creatinine 
(n=48, 18.7%). These patients were in the non-severe 
group, which did not require organ-specific investiga- 
tions. We did the sensitivity analysis by replacing the 
missing data by the mean values calculated from existing 
nonmissing data for each of the corresponding variables. 
When this was done, the discrimination between non-severe 
and severe and fatal was reduced (AuROC =84.41%) but 
was still considered excellent, 19 while the discrimination 
between fatal and non-severe and severe was unchanged. 
However, sensitivity analysis by multiple imputation of 
the missing data yielded almost identical performances 
(data not shown). 

These sensitivity analyses imply that the derived risk- 
scoring algorithm might be successfully and safely applied 
to classify future patients, especially those with severe and 
fatal risk of scrub typhus. The score should help physicians 
in charge determine the need for further investigations 



and start the necessary treatments for those at these risk 
levels. 

The main advantage of the risk-scoring algorithm is 
that it requires routinely available patient information, 
comprising demographic, physical examination, and 
laboratory test data. 17 In endemic areas with restricted 
health resources, 20-22 the score may help inform clinical 
decision making with regard to both investigations and 
treatments. 

In the present study, data on the validation patients were 
collected retrospectively and were based on provisional diag- 
nosis made as part of routine clinical practice, in which scrub 
typhus cases may reflect "suspected" rather than "confirmed" 
cases, by definition. Therefore, prospective validation to patients 
with a theoretically defined diagnosis of scrub typhus 18 would 
be helpful to confirm the algorithm's feasibility for future use 
in routine practice. 

Conclusion 

A risk-scoring algorithm developed previously to classify 
scrub typhus patients into the three severity levels was 
validated using a different set of patients. The algorithm 
demonstrated good performance in this subsequent set of 
patients. The algorithm was clinically acceptable and could 
be applied in routine clinical practice to help identify patients 
with severe prognosis in order to judge the need for more 
intensive investigation and treatments. 
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